HPEm Analysis

In [1]:
from IPython.display import HTML

HTML('''<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
Out[1]:
In [18]:
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import matplotlib as mpl 
import datetime
import math
import scipy as sc
from scipy import stats
from datetime import datetime
from datetime import timedelta
from IPython.display import Image
import pylab
import matplotlib.cm as cm
import json
#to make interactive figures
from ipywidgets import interact
from bokeh.plotting import figure, show, output_notebook
from bokeh.models.widgets import Panel, Tabs
#from bokeh.io import output_file, show
output_notebook()

import sys
sys.path.insert(0, "/Users/jennavergeynst/Applications/TeX/")
sys.path.insert(0, "/Users/jennavergeynst/Documents/AllroundFunctions/")
from plotFunctions import HPEmCompare
BokehJS successfully loaded.
In [3]:
refTags = list(['R1','R2','R3'])
syncTags = list(['S7','S8','S10','S12','S13','S14','S15','S16','S17','S18'])
# S18' and S9 are stations without synctags.
IDlist = refTags + syncTags
In [4]:
# load positioning data of reference and synctags
dataRefSync = {}
for tagID in IDlist:
    filename = '~/Documents/DataHam/VPS/VPS-INBO-01-Results20160304/Positions/TRANSMITTER-'+tagID+'-CALC-POSITIONS.csv'
    temp = pd.read_csv(filename)
    temp['DATETIME'] = pd.to_datetime(temp['DATETIME'])
    dataRefSync[tagID] = temp

dataRefSync_kols = pd.concat(dataRefSync, axis=0)
dataRefSync = pd.concat(dataRefSync, axis = 1)

Some stats

In this section, I calculated some statistics (mean, median, standard deviation, quartiles) for the 3 reference tags and 10 sync tags, for both HPE and HPEm.

In [5]:
dataRefSync.loc[:, (slice(None), 'HPE')].describe()
Out[5]:
R1 R2 R3 S10 S12 S13 S14 S15 S16 S17 S18 S7 S8
HPE HPE HPE HPE HPE HPE HPE HPE HPE HPE HPE HPE HPE
count 7983.000000 8197.000000 7907.000000 7708.000000 7600.000000 7801.000000 7891.000000 7726.000000 7930.000000 7130.000000 8076.000000 7227.000000 7989.000000
mean 1.702956 2.013664 2.308221 1.977128 6.641658 23.506384 2.497440 3.412581 1.767654 95.324741 1.805535 4.080061 1.870635
std 0.128270 5.688344 11.630602 0.765382 81.716792 701.577227 8.481694 1.158919 0.361917 4183.619757 1.225955 3.554346 2.483353
min 1.600000 1.700000 1.600000 1.700000 2.000000 1.900000 1.900000 2.000000 1.700000 2.200000 1.700000 1.800000 1.700000
25% 1.700000 1.800000 1.700000 1.800000 2.600000 2.300000 2.000000 3.000000 1.700000 3.000000 1.700000 2.100000 1.700000
50% 1.700000 1.800000 1.700000 1.800000 2.800000 4.100000 2.000000 3.300000 1.700000 3.100000 1.700000 2.300000 1.700000
75% 1.700000 1.800000 2.300000 1.900000 3.000000 4.800000 2.100000 3.700000 1.800000 3.200000 1.800000 2.700000 1.700000
max 10.100000 446.300000 769.400000 27.500000 4522.000000 59757.300000 386.100000 73.300000 24.300000 274303.900000 68.000000 51.600000 105.200000
In [6]:
dataRefSync.loc[:, (slice(None), 'HPEm')].describe()
Out[6]:
R1 R2 R3 S10 S12 S13 S14 S15 S16 S17 S18 S7 S8
HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm HPEm
count 7958.000000 8197.000000 7905.000000 7708.000000 7600.000000 7801.000000 7891.000000 7726.000000 7930.000000 7130.000000 8076.000000 7227.000000 7989.000000
mean 1.120093 1.090143 5.950310 4.000376 9.764237 6.654365 1.182043 10.671356 2.970391 2.721262 1.073725 20.900996 0.584716
std 1.286035 4.643884 9.550833 6.760660 15.625609 10.257674 1.568217 12.554283 2.655242 4.082697 2.716173 20.909207 0.984493
min 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
25% 0.500000 0.500000 0.800000 1.100000 1.600000 1.000000 0.600000 1.200000 1.300000 1.100000 0.500000 1.100000 0.200000
50% 1.000000 0.700000 1.300000 1.600000 2.700000 2.400000 0.900000 7.600000 2.000000 2.100000 0.900000 12.400000 0.400000
75% 1.400000 1.000000 4.000000 5.100000 11.700000 6.900000 1.400000 16.600000 4.900000 3.000000 1.200000 43.800000 0.800000
max 39.100000 111.600000 91.400000 92.100000 135.300000 138.500000 84.700000 111.500000 47.000000 102.100000 109.500000 130.900000 33.400000

Some transmitter stations have a very high mean HPEm (and 3rd quartile), for instance S12, S15, S7. Might be related to their location close to a wall...

Below you see the study site and the receiver lay-out. Positioning occurs only upstream of the sluice complex. The complex contains 3 sluice locks. At the left bank, upstream of the most southern sluice lock, you can see a side channel. This channel is used for pumping (at low flow) and turbining (at high flow), so the channel intake may also cause some noise and reflections (because it is another concrete construction).

In [7]:
Image("/Users/jennavergeynst/Documents/Analysis/QGis visualisaties/ReceiverArray.png")
Out[7]:

Just an overview of stats for all HPEs and HPEms together.

In [8]:
dataRefSync_kols.loc[:,('HPE','HPEm')].describe()
Out[8]:
HPE HPEm
count 101165.000000 101138.000000
mean 10.835860 5.151051
std 1128.031505 10.695595
min 1.600000 0.000000
25% 1.700000 0.700000
50% 1.900000 1.300000
75% 2.800000 3.100000
max 274303.900000 138.500000

HPEm on reference tags

In [10]:
Image("/Users/jennavergeynst/Documents/Analysis/QGis visualisaties/HPE_R1.png")
Out[10]:

The figure above shows the positions calculated for reference tag 1. Color scale showing red to yellow for high to low HPE. My concern here is that some points with quite high HPEm (far from the actual R1-location), have a low HPE (yellow, so an apparently high position precision). Probably, these erroneous positions cannot be identified as 'bad positions' by HPE, because they might be caused by reflections on the wall or other disturbances. I think these high-HPEm points should be filtered out before constructing any relationship between HPEm and HPE. Do you agree?

In [11]:
Image("/Users/jennavergeynst/Documents/Analysis/QGis visualisaties/HPE_R2.png")
Out[11]:

Same here, even worse HPEm-points still have low (yellow) HPE.

In [12]:
Image("/Users/jennavergeynst/Documents/Analysis/QGis visualisaties/HPE_R3.png")
Out[12]:

Same here. Also, it shows quite a strange pattern of datapoints swarming around in different directions, for all positions which should locate the same reference tag. Maybe here the vicinity of the side channel intake (another concrete box) causes even more noise here. Allthough, for some sync transmitters located further from this side channel intake, the pattern is similar...

In the graph below, you can see the HPEm evolution in time for all ref and sync transmitters. Some really high peaks occur for which I would think they should be filtered out first...

In [35]:
ylim = 140
tab=[0]*len(IDlist)
i=0

for tag in IDlist:

    p = figure(title='Evolution in time for HPEm '+tag,plot_height=400, plot_width=950, 
               y_axis_label='HPEm [m]', x_axis_type="datetime", y_range =(-5,ylim))
    p.line(dataRefSync[tag]['DATETIME'], dataRefSync[tag]['HPEm'])
    tab[i] = Panel(child=p, title=tag)
    i=i+1

tabs = Tabs(tabs=tab)

show(tabs)
In [ ]:
 
In [ ]: